Terminate message queries that exceed a configurable time limit by ramonsmits · Pull Request #5848 · Particular/ServiceControl

ramonsmits · 2026-09-03T10:18:35Z

Customer debug packages showed audit message queries against MessagesViewIndexWithFullTextSearch running for up to ~65 minutes. RavenDB's own Databases.QueryTimeoutInSec (default 300) does not stop them because the server renews the operation deadline while a query keeps making progress, so large scans/sorts run for hours and their sorted results spill into the server's Temp folder, filling the disk.

Query time limit

The message view and search queries now run under a linked CancellationTokenSource that cancels after a configurable time limit, default 1 minute, maximum 1 hour. Cancelling the client request aborts the call to the database server, which does terminate the query server-side. Invalid values fall back to the default.

One QueryTimeLimit helper in ServiceControl.Infrastructure serves all persisters. Its rule is that whatever exception surfaces after the deadline fired is the timeout: RavenDB and Npgsql raise OperationCanceledException, Microsoft.Data.SqlClient raises SqlException("Operation cancelled by user"). The SQL Server path is covered by a test against the real container that slows the command down server-side.

The limit applies to exactly these queries:

Audit RavenAuditDataStore: GetMessages, QueryMessages, QueryMessagesByReceivingEndpoint, QueryMessagesByReceivingEndpointAndKeyword, QueryMessagesByConversationId, read from ServiceControl.Audit/QueryTimeoutInSeconds. QueryAuditCounts (licensing throughput) and QuerySagaHistoryById are not under the limit.
Primary IMessagesViewDataStore on RavenDB, SQL Server and PostgreSQL: GetAllMessages, GetAllMessagesForEndpoint, GetAllMessagesByConversation, GetAllMessagesForSearch, SearchEndpointMessages, read from ServiceControl/QueryTimeoutInSeconds. On SQL Server and PostgreSQL these queries also use the limit as their per-command timeout, so Database/CommandTimeout cannot undercut it.

Other paged index queries (GetFailedMessages, groups, event log, ...) are unchanged.

What a caller sees

A query that runs out of its time is answered with 504 Gateway Timeout and a application/problem+json body whose detail names the setting, on both hosts.
The primary's scatter-gather keeps partial data: a local timeout is absorbed the way a remote failure already was, a remote 504 or error is a missing instance rather than an instance with no data. The response lists every missing instance in the X-Particular-Incomplete-Results header (instanceId:timeout|unavailable|error, exposed through CORS) and carries no ETag. Only when no queried instance answered and one of them timed out does the composite itself answer 504.
The in-process audit counts used for licensing throughput fail when any instance is missing, instead of recording a partial sum as the day's throughput.
The remote instance HttpClient timeout is ServiceControl/QueryTimeoutInSeconds plus 30 seconds instead of the 100 second default, so raising the limit on both instances does not make the primary give up on the audit instance first.

ServicePulse follow-up: the audit list already treats a non-2xx answer (the new 504) as a failed query; reading X-Particular-Incomplete-Results on a 200 is still to be added.

Docs: Particular/docs.particular.net#8486

ServiceControl (Particular/ServiceControl#5848) now answers a partial composite with the X-Particular-Incomplete-Results header naming every instance whose data is missing (instanceId:timeout|unavailable|error), and a fully timed-out query with a 504. All Messages now reads both: a partial page keeps its data but shows a 'Partial results' warning naming each missing instance and its reason, and a 504 gets a precise 'exceeded the ServiceControl query time limit' message instead of the generic failure text. The header parser is shared so the conversation and saga views can adopt it next.

Customer debug packages showed audit message queries against MessagesViewIndexWithFullTextSearch running for up to ~65 minutes. RavenDB's own Databases.QueryTimeoutInSec (default 300) does not stop them because the server renews the operation deadline while a query keeps making progress, so large scans/sorts run for hours and their sorted results spill into the server's Temp folder, filling the disk. The message view and search queries (all messages, per endpoint, search, conversation) now run under a linked CancellationTokenSource that cancels after a configurable time limit, default 1 minute, maximum 1 hour. Cancelling the client request aborts the call to the database server, which does terminate the query server-side. On expiry a TimeoutException names the setting to adjust; invalid values fall back to the default. One QueryTimeLimit helper in ServiceControl.Infrastructure serves every persister, so they share one setting with one behavior, a hard wall-clock deadline per data store call. Its rule is that whatever exception surfaces after the deadline fired is the timeout: RavenDB and Npgsql raise OperationCanceledException, Microsoft.Data.SqlClient raises SqlException("Operation cancelled by user"), which a plain OperationCanceledException catch never sees. A test against the real SQL Server and PostgreSQL containers slows the command down server-side to cover that. - ServiceControl.Audit/QueryTimeoutInSeconds bounds the audit RavenAuditDataStore message view queries. The licensing audit counts and the saga history lookup are not under the limit. - ServiceControl/QueryTimeoutInSeconds bounds the primary instance IMessagesViewDataStore queries on RavenDB, SQL Server and PostgreSQL, which use the same unbounded sorted index query shape. On SQL Server and PostgreSQL these queries also use the limit as their per-command timeout, so Database/CommandTimeout cannot undercut it. Co-authored-by: Mauro Servienti <mauro.servienti@gmail.com>

ServiceControl (Particular/ServiceControl#5848) now answers a partial composite with the X-Particular-Incomplete-Results header naming every instance whose data is missing (instanceId:timeout|unavailable|error), and a fully timed-out query with a 504. All Messages now reads both: a partial page keeps its data but shows a 'Partial results' warning naming each missing instance and its reason, and a 504 gets a precise 'exceeded the ServiceControl query time limit' message instead of the generic failure text. The header parser is shared so the conversation and saga views can adopt it next.

mauroservienti · 2026-09-07T13:52:58Z

@ramonsmits, I have not finished reviewing this yet. However, we also have this CustomCheck that probably needs to be adapted to incorporate the query timeouts. Also, we have this concept of an instance being temporarily unavailable if it fails to respond to the mentioned custom check. When that property is true, the ScatterGather API skips the unavailable instance until it comes back.

ramonsmits · 2026-09-07T15:18:37Z

Looked at CheckRemotes against the query timeouts. The check probes /api, which runs no query, so a remote whose queries time out never fails the health check, and that is intended: a 504 means the remote is up, so TemporarilyUnavailable stays false and the next query asks it again (A_remote_query_timeout_keeps_the_local_data_and_reports_the_remote_instance asserts that).

There was a real interaction though, caused by this PR: the probe used the remote's named HttpClient, whose timeout is now the query time limit. With a short limit a healthy but slow remote would fail the health check, and the failure message would still claim 10 seconds. While in there I also fixed a pre-existing problem: the probe's own timeout never applied, because its cancellation was swallowed by a no-op branch, so a hanging remote passed the check and was never disabled, and a shutdown waited out the full budget.

79d19e9 gives the probe its own budget independent of the query limit, makes a remote that doesn't answer in time fail the check and become temporarily unavailable, and lets a shutdown abort the probe without a verdict. CheckRemotesTests covers the four cases.

Since your last pass I also dropped the 30 second margin (22e6109): the remote client now times out at this instance's limit, so ServiceControl/QueryTimeoutInSeconds bounds the whole composite. An audit instance that is slow, hung, or configured with a larger limit is reported as missing rather than stretching the response. The docs PR is updated accordingly.

ServiceControl (Particular/ServiceControl#5848) now answers a partial composite with the X-Particular-Incomplete-Results header naming every instance whose data is missing (instanceId:timeout|unavailable|error), and a fully timed-out query with a 504. All Messages now reads both: a partial page keeps its data but shows a 'Partial results' warning naming each missing instance and its reason, and a 504 gets a precise 'exceeded the ServiceControl query time limit' message instead of the generic failure text. The header parser is shared so the conversation and saga views can adopt it next.

mauroservienti

This is good to go, @ramonsmits 🚀

ServiceControl (Particular/ServiceControl#5848) now answers a partial composite with the X-Particular-Incomplete-Results header naming every instance whose data is missing (instanceId:timeout|unavailable|error), and a fully timed-out query with a 504. All Messages now reads both: a partial page keeps its data but shows a 'Partial results' warning naming each missing instance and its reason, and a 504 gets a precise 'exceeded the ServiceControl query time limit' message instead of the generic failure text. The header parser is shared so the conversation and saga views can adopt it next.

…data A data store's TimeoutException now becomes a 504 Gateway Timeout with a problem body naming the setting, on both the primary and the audit host, so a caller can tell a timeout from a crash and from an empty result. The primary's scatter-gather no longer hides what went wrong. A local timeout is absorbed the way a remote failure already was, a remote 504 or error is a missing instance rather than an instance with no data, and the response names every missing instance in the X-Particular-Incomplete-Results header (instanceId:timeout|unavailable| error) and carries no ETag. Only when no instance that was asked answered and one of them timed out does the composite itself fail with the timeout. The signal is a header rather than a body field because the composite endpoints return a bare array; an envelope would change the response schema and need a new API for every existing client. A remote's timeout does not cut the slower remotes short: the composite waits for every remote that is still answering and keeps their data. The in-process audit counts used for licensing throughput fail instead of recording a partial sum as the day's throughput. ServiceControl/QueryTimeoutInSeconds bounds the whole composite. The remote instance HttpClient times out at that limit instead of the 100 second default, so an audit instance that is slow, hung, or configured with a larger ServiceControl.Audit/QueryTimeoutInSeconds is reported as missing rather than stretching the response. The audit instance's own limit still ends the query on its side.

…ount The probe used the remote's HttpClient, whose timeout is now the query time limit, so a short limit failed the health check of a remote that was fine. The probe's own 10 second budget never applied: its cancellation was caught by a no-op branch, so a hanging remote passed the check and was never disabled, and a shutdown waited out the full budget. The probe now runs on its own budget regardless of the query time limit; a remote that does not answer in time is disabled and fails the check, and a shutdown aborts the probe without a verdict on the remote.

ServiceControl (Particular/ServiceControl#5848) now tells clients when a composite result is incomplete: a partial answer stays HTTP 200 with the data that did arrive plus an X-Particular-Incomplete-Results header naming every instance that contributed nothing (instanceId:timeout| unavailable|error), and a query nobody answered becomes a 504. All Messages reads both. A partial page keeps its data and shows a warning naming each missing instance and why. The instance id is ServiceControl's base64 API URL, decoded and shown as host and port with the URL as tooltip; the warning says what to do: the rows shown come from the instances that did answer, try again in a moment, or check the listed instances on Platform Health (linked), and when one timed out, that a narrower time range makes the query lighter. The results line presents the total as a floor while results are partial ("of at least 87,421,337"). A 504 gets a precise "exceeded the ServiceControl query time limit" message with the existing one-click narrower-range escape hatches. The state is forgotten with the results when the view is left. The header parser is shared (incompleteResults.ts) so the conversation and saga views can adopt it next.

internalautomation Bot assigned ramonsmits Sep 3, 2026

ramonsmits added the Improvement label Sep 3, 2026

ramonsmits requested a review from mauroservienti September 3, 2026 10:18

ramonsmits added this to the 6.20.0 milestone Sep 3, 2026

ramonsmits mentioned this pull request Sep 3, 2026

Document the ServiceControl QueryTimeoutInSeconds settings Particular/docs.particular.net#8486

Draft

mauroservienti requested changes Sep 3, 2026

View reviewed changes

Comment thread src/ServiceControl.Audit.Persistence.RavenDB/RavenAuditDataStore.cs Outdated

Comment thread src/ServiceControl.Persistence.RavenDB/ErrorMessagesDataStore.cs Outdated

Comment thread src/ServiceControl.Persistence.EFCore/Implementation/MessagesViewDataStore.cs Outdated

This was referenced Sep 3, 2026

Fire a single audit query per query-control change Particular/ServicePulse#3104

Merged

Keep All Messages usable during slow queries Particular/ServicePulse#3105

Open

ramonsmits requested a review from mauroservienti September 3, 2026 16:43

mauroservienti approved these changes Sep 4, 2026

View reviewed changes

ramonsmits marked this pull request as draft September 4, 2026 14:47

ramonsmits force-pushed the ramon/audit-query-timeout branch from b7e02a4 to ec9cf12 Compare September 4, 2026 15:24

ramonsmits requested a review from mauroservienti September 4, 2026 18:32

ramonsmits mentioned this pull request Sep 4, 2026

Surface incomplete scatter-gather results and server query timeouts Particular/ServicePulse#3107

Open

ramonsmits marked this pull request as ready for review September 7, 2026 08:48

ramonsmits force-pushed the ramon/audit-query-timeout branch from ec9cf12 to e393787 Compare September 7, 2026 13:40

mauroservienti reviewed Sep 7, 2026

View reviewed changes

Comment thread src/ServiceControl/Infrastructure/WebApi/HttpResponseExtensions.cs

ramonsmits requested a review from mauroservienti September 7, 2026 15:19

mauroservienti approved these changes Sep 9, 2026

View reviewed changes

ramonsmits added 2 commits September 9, 2026 14:10

ramonsmits force-pushed the ramon/audit-query-timeout branch from 79d19e9 to 276acd7 Compare September 9, 2026 12:11

ramonsmits merged commit 3d92002 into master Sep 9, 2026
68 of 70 checks passed

ramonsmits deleted the ramon/audit-query-timeout branch September 9, 2026 13:19

Navigation Menu

Uh oh!

Terminate message queries that exceed a configurable time limit - #5848

Terminate message queries that exceed a configurable time limit#5848
ramonsmits merged 3 commits into
masterfrom
ramon/audit-query-timeout

ramonsmits commented Sep 3, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mauroservienti commented Sep 7, 2026

Uh oh!

Uh oh!

ramonsmits commented Sep 7, 2026

Uh oh!

mauroservienti left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

ramonsmits commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Query time limit

What a caller sees

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mauroservienti commented Sep 7, 2026

Uh oh!

Uh oh!

ramonsmits commented Sep 7, 2026

Uh oh!

mauroservienti left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ramonsmits commented Sep 3, 2026 •

edited

Loading